Place Search by Audio Signals

ABSTRACT

The present disclosure provides systems and methods that provides users with information pertaining to the audio properties at one or more points of interest. A database associated the audio properties with the points of interest is built using audio input received from devices at the points of interest. The device may determine that audio properties associated with the received audio input. The audio properties may determine a type of background noise and/or a volume of the background noise. If the type of background noise is music, the audio properties may further include a music genre, a title of a song, whether the music is recorded or there is a live band, etc. The audio properties associated with the point of interest may be updated in a database real time.

BACKGROUND

Many devices, including smartphones and smartwatches, provide forgeographic location searching and for querying points of interest. Forexample, a user may provide a geographical location as part of a searchquery for points of interest in or around that geographic location. Theuser may further include, as part of their search query, a type of pointof interest they are searching for, such as a restaurant, an activity,center, a concert venue, movie theater, etc. For example, the user maysearch for “coffee shops in Springfield.” Information regarding pointsof interest matching the search query to some degree are provided to theuser in response to the search query.

BRIEF SUMMARY

One aspect of the disclosure provides for a device for updatinginformation regarding points of interest with determined audioproperties. For example, the device may have one or more memories andone or more processors in communication with the one or more memories.The one or more processors may be configured to receive, from one ormore microphones of a client device, audio input. The one or moreprocessors may be further configured to determine one or more audioproperties based on the received input, receiving location informationassociated with the audio input, associate the determined audioproperties with a particular point of interest based on the receivedlocation information, and update information regarding the particularpoint of interest stored in the one or more memories with the determinedone or more audio properties.

The one or more processors may be further configured to receive a searchquery, identify one or more selected points of interest in response tothe search query, and provide for output at least one of the selectedpoints of interest along with the determined audio properties. Thesearch request may specify at least one of the one or more audioproperties. The one or more audio properties may include at least one ofa music genre, a title of a currently plating song, or a volume level.The volume level may indicate a music playback volume or a backgroundnoise volume. The information regarding the particular point of interestmay be updated in real time. The audio pint may be received from the oneor more microphones without a prompt from a user. The informationregarding the particular point of interest may be part of a mappingdatabase.

Another aspect of the disclosure provides for a method for updatinginformation regarding points of interest with determined audioproperties. The method includes receiving, using one or more processors,audio input. The method includes determining, by the one or moreprocessors based on the received audio input, one or more audioproperties, receiving, by the one or more processors, locationinformation associated with the audio input, associating, by the one ormore processors based on the received location information, thedetermined audio properties with a particular point of interest, andupdating, by the one or more processors, information regarding theparticular point of interest with the determined one or more audioproperties.

Yet another aspect of the disclosure provides for a non-transitorycomputer-readable medium storing instructions executable by a processorfor performing a method comprising receiving, from one or moremicrophones, audio input, determining, by one or more processors basedon the received audio input, one or more audio properties, receiving, bythe one or more processors, location information associated with theaudio input, associating, by the one or more processors based on thereceived location information, the determined audio properties with aparticular point of interest, and updating, by the one or moreprocessors, information regarding the particular point of interest withthe determined one or more audio properties.

Another aspect of the disclosure provides for a system comprising one ormore memories storing a plurality of points of interest in associationwith one or more audio properties and one or more processors incommunication with the one or more memories. The one or more processorsmay be configured to receive, from at least one client computing device,a search query for one or more points of interest. The search query mayinclude one or more audio properties associated with the one or morepoints of interest. The one or more processors may be configured toidentify, in response to the search query, one or more selected pointsof interest stored in the one or more memories. The audio propertiesassociated with the one or more selected points of interest maycorrespond with the one or more audio properties included in the searchquery. The one or more processors may be configured to provide foroutput at least one of the selected points of interest along with thecorresponding one or more audio properties.

Yet another aspect of the disclosure provides for a method for providingoutput at least one point of interest along with corresponding audioproperties. The method may include receiving, from at least one clientcomputing device, a search query for one or more points of interest. Thesearch query may include one or more audio properties associated withthe points of interest. The method may include identifying, in responseto the search query, one or more selected points of interest. The audioproperties associated with the one or more selected points of interestmay correspond with the one or more audio properties included in thesearch query. The method may include providing for output at least oneof the selected points of interest along with the corresponding one ormore audio properties.

Another aspect of the disclosure provides for a non-transitorycomputer-readable medium storing instructions executable by a processorfor performing a method comprising receiving, from at least one clientcomputing device, a search query for one or more points of interest. Thesearch query may include one or more audio properties associated withthe one or more points of interest. The method may include identifying,by one or more processors in response to the search query, one or moreselected points of interest. The audio properties associated with theone or more selected points of interest may correspond with the one ormore audio properties included in the search query. The method mayinclude providing, by the one or more processors, for output at leastone of the selected points of interest along with the corresponding oneor more audio properties.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example device according toaspects of the disclosure.

FIG. 2 is a functional diagram of an example system according to aspectsof the disclosure.

FIG. 3 is a pictorial diagram illustrating an example use of the deviceaccording to aspects of the disclosure.

FIGS. 4A-4C are a series of example interfaces corresponding to theexample use of FIG. 3.

FIG. 5 is a pictorial diagram illustrating another example use of thedevice according to aspects of the disclosure.

FIGS. 6A-6C are a series of example interfaces corresponding to theexample use of FIG. 5.

FIG. 7 is a flow diagram illustrating a method of associating audioproperties with a point of interest according to aspects of thedisclosure.

FIG. 8 is an example interface illustrating a query for a particularpoint of interest according to aspects of the disclosure.

FIG. 9 is an example interface illustrating information regarding thepoints of interest according to aspects of the disclosure.

FIG. 10 is an example interface illustrating information regarding aparticular point of interest according to aspects of the disclosure.

FIG. 11 is a flow diagram illustrating a method of searching for a pointof interest based on audio properties according to aspects of thedisclosure.

DETAILED DESCRIPTION

The present disclosure relates to a system that provides users withinformation pertaining to the audio properties at one or more points ofinterest. A database associating the audio properties with the points ofinterest is built using audio input received from devices at the pointof interest. For example, such devices include one or more microphonesthat detect background noise, such as people talking, music playing inthe background, etc. A type of the background noise and/or a volumelevel of the background noise may be determined from the audio input,and such information may be associated with further information, such asaddress, operating hours, etc., regarding the point of interest in thedatabase. For example, the particular point of interest may be part of amapping database. In some examples, the audio properties may be updatedin the mapping database in real time. Accordingly, search queries fromuser devices may identify one or more points of interests based on theaudio properties. For example, a user may submit a search query for aquiet cafè or a bar that plays country music and receive relevantresults.

The audio properties may include, for example, the type of backgroundnoise, volume level, or other characteristics of the received audio. Thetype of background noise may classify the noise as music, talking,traffic, construction, or any of a variety of other types of noise.According to some examples, where the noise is music, the audioproperties may further specify a music genre, a title of a song, whetherthe music is recorded or there is a live band, etc.

FIG. 1 provides an example block diagram illustrating components of thedevice. As shown, the device 100 includes various components, such asone or more processors 102, memory 104, and other components typicallypresent in microprocessors, general purpose computers, or the like.Device 100 also includes input 110, microphones 112, and a locationdeterminer such as a global positioning system (“GPS”) sensor 114.

The one or more processors 102 may be any conventional processors, suchas commercially available microprocessors. Alternatively, the one ormore processors may be a dedicated device such as an applicationspecific integrated circuit (ASIC) or other hardware-based processor.Although FIG. 1 functionally illustrates the processor, memory, andother elements of device 100 as being within the same block, it will beunderstood by those of ordinary skill in the art that the processor,computing device, or memory may actually include multiple processors,computing devices, or memories that may or may not be stored within thesame physical housing. Similarly, the memory may be a hard drive orother storage media located in a housing different from that of device100. Accordingly, references to a processor or computing device will beunderstood to include references to a collection of processors orcomputing devices or memories that may or may not operate in parallel.

Memory 104 may store information that is accessible by the processors102, including instructions 106 that may be executed by the processors102, and data 108. The memory 104 may be of a type of memory operativeto store information accessible by the processors 102, including anon-transitory computer-readable medium, or other medium that storesdata that may be read with the aid of an electronic device, such as ahard-drive, memory card, read-only memory (“ROM”), random access memory(“RAM”), optical disks, as well as other write-capable and read-onlymemories. The subject matter disclosed herein may include differentcombinations of the foregoing, whereby different portions of theinstructions 106 and data 108 are stored on different types of media.

Data 108 may be retrieved, stored or modified by processors 102 inaccordance with the instructions 106. For instance, although the presentdisclosure is not limited by a particular data structure, the data 108may be stored in computer registers, in a relational database as a tablehaving a plurality of different fields and records, XML documents, orflat files. The data 108 may also be formatted in a computer-readableformat such as, but not limited to, binary values, ASCII or Unicode. Byfurther way of example only, the data 108 may be stored as bitmapscomprised of pixels that are stored in compressed or uncompressed, orvarious image formats (e.g., JPEG), vector-based formats (e.g., SVG) orcomputer instructions for drawing graphics. Moreover, the data 108 maycomprise information sufficient to identify the relevant information,such as numbers, descriptive text, proprietary codes, pointers,references to data stored in other memories (including other networklocations) or information that is used by a function to calculate therelevant data.

The instructions 106 can be any set of instructions to be executeddirectly, such as machine code, or indirectly, such as scripts, by theprocessor 102. In that regard, the terms “instructions,” “application,”“steps,” and “programs” can be used interchangeably herein. Theinstructions can be stored in object code format for direct processingby the processor, or in any other computing device language includingscripts or collections of independent source code modules that areinterpreted on demand or compiled in advance. Functions, methods androutines of the instructions are explained in more detail below.

The device 100 may further include an input 110. The input 110 may be,for example, a touch sensor, dial, button, or other control forreceiving a manual command. The input 110 may, in some examples, be amicrophone. The device 100 may also include an output 116. The output116 may be, for example, a speaker.

Device 100 may include at least one microphone 112. The microphones 112may be able to receive audio input. The audio input may includebackground noise. Audio properties of the background noise may bedetermined. The audio properties may include, for example, the type ofbackground noise, such as whether the background noise is music, peopletalking, street noise such as traffic or construction, etc. When thebackground noise is music, the audio properties may further determinethe genre, title of the song, the artist, etc.

Device 100 may further include at least one GPS sensor 114. The GPSsensors 114 may provide a rough indication as to the location of thedevice. The determined audio properties may be used to refine thelocation indicated by the GPS sensors or vice versa.

It should be understood that the device 100 may include other componentswhich are not shown, such as a battery, charging input for the battery,signal processing components, etc. Such components may also be utilizedin execution of the instructions 106.

FIG. 2 illustrates an example system 200 in which the features describedherein may be implemented. It should not be considered limiting thescope of the disclosure or usefulness of the features described herein.In this example, system 200 may include a plurality of devices 202, 212,222, 232, users 204, 214, 224, 234, server computing device 270, storagesystem 260, and network 250. For purposes of ease, the collection ofdevices 202, 212, 222, 232 or a single device will be referenced asdevice(s) 202. Further, also for purposes of ease, the group of users204, 214, 224, 234 and a single user will be referenced as user(s) 204.

Each device 202 may be a personal computing device intended for use by arespective user 204 and have all of the components normally used inconnection with a personal computing device, as described above withrelationship to device 100, including a one or more processors (e.g., acentral processing unit (CPU)), memory (e.g., RAM and internal harddrives) storing data and instructions, a display (e.g., a monitor havinga screen, a touch-screen, a projector, a television, or other devicesuch as a smart watch display that is operable to display information),and user input devices (e.g., a mouse, keyboard, or touchscreen). Thedevices may also include one or more microphones. The microphone mayreceive audio input. For example, the microphone may receive a verbalcommand from the user. In some examples, the microphone may receiveaudio input at one or more points of interest. The audio input mayinclude background noise, such as the lull of conversations in thebackground, music, construction, traffic, etc. The devices 202 may alsoinclude a camera, speakers, a network interface device, and all of thecomponents used for connecting these elements to one another. Thedevices 202 may further include the image sensors. The image sensors maycapture features and/or object of a plurality of points of interest.Device 202 may be capable of wirelessly exchanging and/or obtaining dataover the network 250.

Although the devices 202 may each comprise mobile computing devicescapable of wirelessly exchanging data with a server over a network suchas the Internet, they may alternatively comprise a full-sized personalcomputing device. By way of example only, devices may be mobile phonesor devices such as a wireless-enabled PDA, a tablet PC, a wearablecomputing device (e.g., a smartwatch, headset, smartglasses, virtualreality player, other head-mounted display, etc.), or a netbook that iscapable of obtaining information via the Internet or other networks.

The devices 202 may be at various nodes of a network 250 and capable ofdirectly and indirectly communicating with other nodes of network 250.Although four (4) devices are depicted in FIG. 2, it should beappreciated that a typical system 200 can include one or more devices,with each computing device being at a different node of network 250. Thenetwork 250 and intervening nodes described herein can be interconnectedusing various protocols and systems, such that the network can be partof the Internet, World Wide Web, specific intranets, wide area networks,or local networks. The network 250 can utilize standard communicationsprotocols, such as WiFi, that are proprietary to one or more companies.Although certain advantages are obtained when information is transmittedor received as noted above, other aspects of the subject matterdescribed herein are not limited to any particular manner oftransmission.

In one example, system 200 may include one or more server computingdevices having a plurality of computing devices, e.g., a load balancedserver farm, that exchanges information with different nodes of anetwork for the purpose of receiving, processing and transmitting thedata to and from other computing devices. For instance, one or moreserver computing devices 270 may be a web server that is capable ofcommunicating with the one or more devices 202 via the network 250. Inaddition, server computing device 270 may use network 250 to transmitand present information to a user 204 of one of the other devices 202.Server computing device 270 may include one or more processors 272,memory 274, instructions 276, and data 278. These components operate inthe same or similar fashion as those described above with respect todevice 100.

Storage system 260 may store various types of information. For instance,the storage system 260 may store information about points of interest,such as publically accessible ratings, map data, etc. The storage system260 may store map data. The map data may include, for instance,locations of points of interest. According to some examples, the mapdata may include audio properties associated with locations of points ofinterest. This information may be retrieved or otherwise accessed by aservice computing device, such as one or more server computing devices270, in order to perform some or all of the features described herein.

FIG. 3 illustrates an example where a plurality of users 304, 314 are ata particular point of interest. The point of interest may be arestaurant, activity center, concert venue, retail store, landmark,museum, etc. Each user's device 302, 312 may detect audio properties ofthe point of interest if the user has enabled this function. The devicemay be configured to only receive background noise as audio input whenauthorized by the user. For example, the user may enable or disable abackground noise detection feature. The user may further adjust thesettings on the device to restrict when background noise may be receivedas audio input. For example, background noise detection may be limitedto certain public locations or other conditions, such as when aparticular application is in use, a predetermined number of other deviceusers are within range, a particular type of reader or scanner isdetected, etc.

As shown in FIG. 3, the users 304, 314, 324 may be within cafè 300. Thedevices 302, 312 may be similar to device 100 described above. Forpurposes of ease, devices 302, 312 may be collectively referred toherein as device 302.

There may be background noise, including music 320, within cafè 300.While music 320 is shown as being played through a speaker, music 320may include a DJ, live music, radio, etc. The background noise may alsoinclude people talking, outside noise such as traffic and construction,or any variety of other noises.

The device 302 may detect the background noise, such as through themicrophone, and determine a volume level of the background noise. Forexample, the device 302 may determine whether the background noisevolume exceeds a predetermined threshold to be considered loud. Thedevice 302 may include a plurality of predetermined thresholds orintervals to determine whether the background noise volume is loud,conversational, quiet, etc. For example, if the background noise volumeis below 50 dB, the background noise volume may be determined to bequiet. If the background noise volume is between 50-70 decibels (dB),the background noise volume may be determined to be conversational orallow for conversation. If the background noise volume is between 70-90dB, the background noise volume may be determined to be noisy, such aheavy traffic or a noisy restaurant. If the background noise volumelevel is above 90 dB, the background noise volume may be determined tobe loud, such as a live sporting event or music. The ranges describedherein are merely examples. The volume ranges for what constitutesnoisy, conversational, or quiet may change as the device 302 receivesverification or feedback from users about the noise levels. For example,the device 302 may ask verification questions pertaining to the receivedaudio input. The verification questions may ask whether the backgroundnoise volume level is loud. If the user answers yes, the device maydetermine the background noise volume level, in decibels, and use thatvolume level to create or update the ranges. Moreover, theclassifications provided herein are merely exemplary and may havedifferent labels and may change based on the feedback from users.

The device 302 may determine the type of background noise based on theaudio input. For example, the device 302 may determine whether thereceived audio input is the background chatter of people talking, musicplaying, street noise, etc. The device 302 may determine that thereceived audio input is a combination of music 320, background chatter,street noise, etc. According to some examples, the device 302 maydetermine additional audio properties for each of the determined typesof background noise. The additional audio properties may include, forexample, a volume level, pitch, tone, frequency, intensity, direction,etc. If the device determines that the received audio is music, thedevice may determine a genre, artist, song title, etc.

In examples where the background noise is the lull of people chattering,the device may determine a volume level associated with the backgroundchatter. In some examples where the background noise is the lull ofpeople chattering, the device 302 may determine a pitch, tone, etc.associated with the chatter. A user may disable receiving backgroundnoise as audio input at any time. Similar to determining whether thebackground noise volume is loud, the device 302 may determine whetherthe volume of the chatter exceeds a predetermined threshold to beconsidered loud. The device 302 may include a plurality of predeterminedthresholds or intervals to determine whether the volume of the chatteris loud, average, quiet, etc. A quiet chatter volume may indicate thatmeetings or other conversations can easily occur. In some examples, thedevice 302 may determine whether people are talking loudly over otherbackground noise, such as over music 320.

In examples where the background noise is music playing, such as music320, the device may determine additional audio properties, such as avolume level, genre, title of the song, etc. For example, the device 302may determine a volume level associated with the music 320. Similar todetermining whether the background noise volume is loud, the device 302may determine whether the music volume exceeds a predetermined thresholdto be considered loud. The device 302 may include a plurality ofpredetermined thresholds or intervals to determine whether the musicvolume is loud, average, quiet, etc. According to some examples, thedevice 302 may determine other audio properties for music 320. Thedevice 302 may determine a music genre, a title of the song playing,whether the music is recorded or whether there music 320 is a live band,etc.

The audio properties may further be associated with timestamps. In thisregard, a particular point of interest may be associated with multipledifferent audio properties at different points in time. For example,while a coffee shop may be a quiet place for working during the day, itmay also be a lively spot with loud folk music in the evening.

Location information may be received and associated with the audioinput. For example, the GPS sensors may provide an indication as to thelocation of the device 302. The GPS data that provides the indication ofthe location of the device 302 may be used to determine a point ofinterest associated with that location. In some examples, informationfrom the sensors and/or from interactions with devices such as wirelessbase stations or wireless access points, RFID readers, etc., can beemployed by the position determination module to calculate or otherwiseestimate the physical location of device 302. Other examples of locationdetermination systems may determine location based on images ofgeographic objects such as landmarks, semantic indicators such as lightor noise level, etc. While these are a few examples, it should beunderstood that any of a variety of location determination techniquesmay be used.

Location information is only provided when authorized by the user, suchas when the device is configured to allow for sharing of locationinformation. The user may enable or disable providing location, and maylimit the recipients of such information and/or the applications or useof the location information. Moreover, the location data excludes anypersonal information. According to some examples, a user may adjustsettings on the device to enable providing location information when thedevice is in particular geographical areas, at certain times, orconditions, such as when the device is within Bluetooth range of apredetermined number of other user devices. The user may set the deviceto automatically disable providing location information in particularlocations, such as familiar locations.

The audio input may be received by the device, such that the device maydetermine the one or more audio properties. The one or more determinedaudio properties may be received at the servers. According to someexamples, the audio input may be received at the servers from a numberof devices which have authorized sharing of detected background noiseinformation. The user can enable or disable receiving background noiseas audio input, and the device may only receive background noise asaudio input if the user has enabled this feature. According to someexamples, a user may adjust settings on the device to restrict sharingto particular geographical areas, particular audio properties or asubset of audio properties, times, or conditions, such as when thedevice is within Bluetooth range of a predetermined number of other userdevices. Moreover, the user may set the one or more microphones toautomatically disable in particular locations. According to someexamples, the user may set the device to automatically disable receivingbackground noise as audio input in particular locations, such asfamiliar locations. The user may further configure the device to receiveaudio input without further active input from the user.

The device may ask the user to verify or provide feedback regarding thereceived audio input. For example, the device may ask a series ofverification questions regarding the location information, volumelevels, type of background noise, etc. In examples where the backgroundnoise includes music, the verification questions may be about the genre,artist, song title, etc. of the music playing. The questions are used toverify the audio properties associated with the point of interest beforeupdating the audio properties associated with the point of interest.

FIGS. 4A-4C illustrate examples where the user may be asked to verifythe audio properties of the audio input received by the device. Forexample, the device may display a series of prompts or questions to theuser. The questions may ask the user to verify the location, the musicgenre, whether the environment is noisy or quiet, whether there is alive music or a DJ, whether the background noise is loud, whether themusic is loud, etc. The device may determine, based on the user' sengagement with the prompts, whether to ask more questions. In someexamples, the device may look to a user's past cooperation and/orparticipation with providing feedback for other applications todetermine whether to ask a series of questions verifying the audioproperties.

As shown in FIG. 4A, device 402 may display a first question 430 seekingfeedback. The first question may ask to verify the point of interest.For example, question 430 asks the user to confirm whether the point ofinterest is a cafè. The user may answer yes 440 or no 450. This mayverify the location information to be associated with the audio input.In some examples, there may be an option to “end” 460. The user mayselect end 460 if the user does not want to answer any verificationquestions.

FIG. 4B illustrates a second question. The second question 432 may beasked based on receiving a response to the first question 430. Thesecond question 432 may ask about the overall background volume level.The background volume level may include street noise, chatter, music,etc. The user may answer question 432 with a yes 442, no 452, or end462. In some examples, the answer may be a ranking scale that the usercan provide a rank between 1-10, with 1 being the quietest and 10 beingthe loudest. In some examples, the scale may be between 1-5. The scalemay be a sliding scale, such that the user does not have to select onenumber but, instead, slides a marker along a scale to indicate thebackground volume level. The user may select end 462 if the user doesnot want to answer additional verification questions

FIG. 4C illustrates a third question that may be based on the secondquestion 432. The second question 432 may ask about the background noisevolume level. The background noise may include music. In some examples,based on the answer to question 432, the third question 434 may askwhether the music volume is loud. The user may select yes 444, no 454,or end 464. In some examples, the user may answer based on a rankingscale or a sliding scale to denote whether the music is loud or not. Ifthe user selects end 464 no further questions will be asked.

Based on the answers provided by the user in response to theverification questions, the audio properties of the point of interestmay be updated in real time. For example, information regarding theparticular point of interest may include indications pertaining to thetype of background noise in real-time. If the type of background noiseis music, the information regarding the particular point of interest mayinclude indication of the music genre, the title of a song playing atthe time audio input is received, artist, volume level of the music,etc. As one example, the information regarding a particular point ofinterest may indicate that classic rock is playing at a moderate volumelevel at the time the audio input is received.

In some examples, real-time audio properties may be compared tohistorical audio properties, or historical and real-time audioproperties may be combined in a statistical model to produce a betterestimate. For example, an evaluation period such as a week, a month,etc. may be used. The real-time audio properties may be provided fordisplay in relation to the historical data for the given point ofinterest.

FIG. 5 illustrates another example where a plurality of users are in aparticular point of interest and their respective devices may bedetecting audio properties of the background noise. For example, users304, 314, 324 may be within a bowling alley 500. Users 504, 514, 524 mayeach have a device 502, 512, respectively. Devices 502, 512 may besimilar to device 100 described herein. For purposes of ease, device502, 512 will be collectively referred to herein as device(s) 502.

Within the bowling alley 500, there may be background noise. Thebackground noise may include the sound of voices, the lull ofconversations, or chatter, the sound of the bowling ball hitting thepins, music 520, etc. Music 520 may be live music, a DJ, radio (i.e.different music tracks mixed together so that there is no gap betweenthem), music from a jukebox, etc.

Devices 502 may receive the background noise, including music 520, asaudio input. The device 502 may determine a type or types of backgroundnoise. For example, the device may determine that there areconversations happening, music is playing, and there is some noiseassociated with the activity in the particular point of interest. Inbowling alley 500, the noise associated with the activity may be thesound of the bowling ball hitting the pins. According to some examples,the device 502 may determine additional audio properties for each of thedetermined types of background noise. The additional audio propertiesmay include an overall background noise volume level, a volume level forchatter, a volume level for music, a genre of music, etc.

In examples where the device 502 determines that the background noiseincludes the sound of voices or chatter at the particular point ofinterest, the device 502 may determine a volume level associated withthe chatter. According to some examples, the device 502 may determinethat people in the bowling alley 500 are talking loudly. People in thebowling alley may have to talk loudly to be heard over the music 520and/or the sound of the ball hitting the pins. For example, on a weekendnight, such as Friday or Saturday night, a live band may playing at thebowling alley 500 with the volume on the speakers turned all the way up.The music 520 may be very loud causing people to have to talk very loudto be heard over music 520. In some examples, the device 502 maydetermine other audio properties associated with the chatter. Forexample, the device 502 may determine a pitch, tone, frequency, etc.associated with the chatter.

During some periods of time, the device 502 may determine that people inthe bowling alley 500 are talking quietly. In such an example, thedevice 502 may determine that the other types of background noise, suchas music 520, are also quiet. For example, during the day on a weekday,the music 520 may be a radio station playing in the background as thebowling alley 500 may not be as busy.

In examples where the background noise includes music 520, the devicemay determine a volume level associated with the music 520. For example,the device 502 may classify the volume level of the music as loud,average, or quiet, etc. Loud music may be music 520 that is played at avolume level above 100 dB. Average music may be music 520 that is playedat a volume level between 60 dB to 100 dB. Quiet music may be music 520that is played at a volume level below 60 dB. These ranges are merelyexamples and may depend on the particular point of interest. Forexample, if a user is at a concert, a loud volume level may be musicthat is played at a volume level above 120 dB and a quiet volume levelmay be music that is played at a volume level below 80 dB. Thus, theranges and labels are merely examples and may change or be calleddifferent things.

According to some examples where the background noise includes music520, the device 502 may determine a music genre, a title of the songplaying, the name of the artist, whether the music 520 is live, on theradio, or being played by a DJ, etc. The determined audio propertiesassociated with the point of interest may be shared with or sent to aserver to update information pertaining to the particular point ofinterest. For example, if the device determines that the bowling alleyis playing pop music by a particular artist, those audio properties maybe used to update the audio properties associated with the bowling alleyin real time.

The audio properties may be associated with a time stamp and particularplace of interest. For example, as the device 502 determines the audioproperties, the audio properties may be associated with the time theywere collected and where they were collected. Audio propertiesassociated with the background noise in the bowling alley 500 on aTuesday afternoon may be different than the audio properties associatedwith the background noise in the bowling alley 500 on a Friday night.For example, on a Tuesday afternoon, the background noise of the bowlingalley 500 may have a quieter overall background noise volume level. Thismay allow for a meeting or conversation to occur within the bowlingalley 500 and/or while bowling. The music 520 may be a classic rockradio station played at a moderate or average volume level. On a Fridaynight, the background noise of the bowling alley 500 may have a loudoverall background noise volume level. The loud background noise volumemay be due to a live band or a DJ playing at the bowling alley 500.Thus, music 520 may be very loud. The loud music 520 may deter peoplefrom hosting meetings or having conversations at the bowling alley 500on a Friday night.

As described previously herein, the determined audio properties of thereceived audio input may be received at a server from a plurality ofdevices that have previously authorized the sharing of the determinedaudio properties. The user may enable or disable the receivingbackground noise as audio input and the determining of audio propertiesof the received audio input. The determined audio properties may beaggregated and used to build a model of the audio properties associatedwith the point of interest. The determined audio properties may begrouped based on the associated time stamp or the time the audioproperties were determined. Once the model is built, subsequentlyreceived audio properties may be used to update the model. As such, whensubsequent search queries are received, the information about the audioproperties associated with particular points of interest may be providedto other users in response.

FIGS. 6A-6C illustrate examples of questions that may be asked to verifythe location information and/or audio properties of the received audioinput. As described herein, the questions may ask the user to confirmthe particular point of interest, the music genre, whether the music islive, the background noise volume level, the music volume level, etc.The device may determine, based on the user's engagement with theprompts, whether to ask more questions. In some examples, the device maylook to a user's past cooperation and/or participation with providingfeedback for other applications to determine whether to ask a series ofquestions verifying the audio properties.

FIGS. 6A-6C may ask verification questions regarding the audio inputreceived while at the bowling alley as compared to FIGS. 4A-4C which mayask verification questions regarding the audio input received while atthe cafè. The questions that may be asked may be based on the type ofbackground noise detected, the volume of the background noise, whetherthe background noise is chatter or music, etc.

As shown in FIG. 6A, device 602 may display a first question 630 seekingfeedback regarding the audio input received while at the bowling alley.The first question 630 may ask the user to verify the point of interest.For example, question 630 may ask the user to confirm whether the pointof interest is a bowling alley. The user may answer question 630 with ayes 640, no 650, or end 660. This may verify the location information tobe associated with the audio input. If the user selects end 660, nofurther questions will be asked.

FIG. 6B illustrates a second question. The second question 632 may beasked as a follow up to the first question 630 if the user answered thefirst question 630.

The second question 632 may ask the user to verify audio propertiesassociated with the received audio input. Question 632 may be asked toconfirm or determine the genre of music playing at the point ofinterest. For example, the second question 632 may ask the user whetherthe music playing in the background is pop music. The user may selectyes 642, no 652, or end 662 in response to question 632. If the userselects no 652, subsequent questions may be asked to determine thecorrect genre of music. If the user selects end 662, the device may notask additional questions.

FIG. 6C illustrates a third question that may be based on the secondquestion 632. The third question 634 may ask about additional audioproperties about the background music received as audio input. Forexample, the third question 634 may ask whether the music volume isloud. The user may provide a response on a sliding scale. For example,the user may adjust indicator 666 along a scale of 1 to 10, with 1 beingthe quietest and 10 being the loudest. In some examples, the scale maybe from 1 to 5. In other examples, the sliding scale may be labeled withquiet and loud instead of numbers. Thus, the scale of 1 to 10 is just anexample and is not meant to be limiting. After the user adjustsindicator 666 along the scale, the user may select “ok” 664 to confirmthe answer. The user may select end 664 to have no further questions.

The answers provided by the user for questions 630, 632, 634 may be usedto update the model of audio properties associated with the point ofinterest in real time. In some examples, the real-time audio propertiesassociated with the point of interest may be compared to other oradditional real-time audio properties collected by other users. Forexample, a plurality of users may be asked to confirm whether the genreof music currently playing is rock. If more than one user confirms thatthe genre of music is rock, the server may update the audio propertiesassociated with the point of interest to show that rock music is knownto be played at that time, day, etc. A comparison of verificationanswers from a plurality of users may be used as an additionalverification measure. For example, the servers may receive conflictinganswers to the verification questions. In such an example, a first usermay answer a verification question finding the background noise level ofa point of interest to be quiet while a second user may answer averification question finding the background noise level of the samepoint of interest to be loud. When the server receives conflictinganswers for the same point of interest, the server may not update theaudio properties associated with the point of interest.

FIG. 7 illustrates an example method for updating information withdetermined audio properties associated with a particular point ofinterest. For example, in block 710, a device may receive audio input.The audio input may include background noise. The background noise mayinclude conversations, street noise such as traffic or construction,music, etc.

In block 720, the device may determine one or more audio propertiesbased on the received audio input. For example, audio properties mayinclude the type of background noise, volume level, or othercharacteristics of the received audio. The type of background noise mayclassify the noise as music, talking, traffic, construction, or any of avariety of other types of noise. According to some examples, where thenoise is music, the audio properties may further specify, a music genre,a title of a song, whether the music is recorded or there is a liveband, etc. In some examples, where the noise is background chatter, theaudio properties may further specify a volume level of the chatter.

In block 730, the device may receive location information associatedwith the audio input. For example, the device may use GPS sensors orother positioning elements to provide an indication as to the locationof the device. In some examples, information from the sensors and/orfrom data received or determined from remote devices (e.g., wirelessbase stations or wireless access points), can be employed by theposition determination module to calculate or otherwise estimate thephysical location of device 302. Other examples of locationdetermination systems may determine location based on images ofgeographic objects such as landmarks, semantic indicators such as lightor noise level, etc. The location of the device may be used to determinea particular point of interest.

In block 740, the received audio input and the determined audioproperties may be associated with a particular point of interest. Forexample, the audio input may be received at a certain geographiclocation at which a particular point of interest is located. Thedetermined audio properties for the audio input may be associated withthe point of interest in which the audio input was received. Forexample, the determined audio properties may be stored in a databasealong with other information pertaining to a plurality of points ofinterest. The other information for a particular point of interest mayinclude, for example, business name, location, operating hours, type ofestablishment, menu, website, etc. The information, including thedetermined audio properties, may be provided in response to a searchquery. In some examples, the determined audio properties may beassociated with certain time intervals of the operating hours, abusyness indicator, etc.

In block 750, information regarding the particular point of interest maybe updated. For example, the information regarding the particular pointof interest may be updated with the determined audio properties. Theinformation may be updated in real time. In some examples, the updatedinformation may provide information as to when the background noiselevel is the loudest. According to some examples, the updatedinformation may provide information regarding what type of music isplayed on each day. For example, the updated information may provide auser information that on Monday country music is played, on Tuesday popmusic is played, on Wednesday classic rock is played, on Thursday thereis live music, etc. The updated information may allow users to identifyparticular points of interest based on the determined audio properties.

Once a model of the audio properties associated with a point of interestis built, a user may search the model. FIG. 8 illustrates an exampleinterface of a search query for finding or determining a plurality ofpoints of interest based on audio properties associated with thosepoints of interest. Device 800 may include a display 820, output 816,input 810 and at least one microphone 812. Display 820 may include asearch bar 824 and a map section 822. Map section 822 may include aposition identifier 826 showing the current location of the device 800.

Display 820 may further include a plurality of search options including“explore” 830, “commute” 832, “for you” 834, and “audio” 836. “Explore”830 provides a plurality of suggestions based on the location indicatedby position identifier 826 of the device 800 or any selected geographiclocation. For example, selecting “explore” 830 may provide suggestionssuch as cake shows, family restaurants, activities, golf courses, etc.around the location indicated by position identifier 826 of the device800 or any other geographic location searched by the user. “Commute” 832may check the traffic between the location 826 of the device 800 and apreset address set by the user. The preset address may be the address ofthe user's location of work. “For you” 834 may provide suggestionsregarding points of interest around the location indicated by positionidentifier 826 of the device 800 that are open for business. Whenselecting “for you” 834, the suggestions may include restaurants,events, businesses, etc.

“Audio” 836 may allow the user to search for locations based on audioproperties associated with the points of interest. The user may searchbased on the type of background noise, such as whether there isconversation and/or music, the volume level of the background noise, thevolume level of conversation, the volume level of the music, whetherthere is other noise in the background such as construction or trafficnoise. The user may search particular audio properties with respect tothe type of background noise. For example, the user may want to find alocation that will allow for conversation that can be heard over thebackground noise. Thus, the user may use the “audio” 836 search to finda place with quiet background noise. In some examples, the user may wantto find a location that has a live band playing on a Friday night. Theuser may be able to refine the search to find only points of interestthat have audio properties indicating that a live band plays on Fridaynights.

Allowing users to search and identify points of interest based on theassociated audio properties may help find a place best suited for theirneeds. For example, people with high frequency hearing loss may havedifficulty having conversations when there is relatively high backgroundnoise level, and therefore may prefer to identify establishments withlow background noise levels. Additionally, people with reduced hearingsensitivity who do not want to have a conversation may preferestablishments with higher volumes of music. As such, allowing users toidentify establishments with high music volumes would benefit thiscategory of person.

FIG. 9 illustrates an example of how search results based on audioproperties may be displayed. Device 900 may be similar to device 800.For example, device 900 may include a display 920, output 916, input 910and at least one microphone 912. These features may be similar to thosediscussed herein with respect to device 100.

The device 900 may provide search results by overlaying the informationon the screen, or display 920. The information may be provided in avariety of different ways, such as a list, a map, annotations to a map,etc. The information may be ordered based on most relevant searchresults, distance to the user, etc. According to other examples, theinformation may be output as audio data through one or more speakers, oroutput 916, of device 900 or accessories paired with device 900.

The plurality of search results may be provided as a detailed list. Eachof the points of interest may be provided as an individual place card940, 942, 944, or button, on display 920. For example, the most relevantpoint of interest based on the audio properties may be a restaurant,shown as place card 940. Place card 940 may include information aboutthe restaurant, including the distance from the determined location ofdevice 900, the music genre, and the background noise volume level. Insome examples, the interactive place card 940 may allow for quickactions relevant to the point of interest. For example, the system maybe integrated with services for the points of interest to allow forquick actions. For a restaurant, the quick action may allow a user tomake a reservation. The second most relevant point of interest based onaudio properties may be a coffee shop, shown as place card 942. Theplace card 942 may indicate that the music genre that is currentlyplaying and the volume level of the background noise based on associatedaudio properties. A quick action for place card 942, the coffee shop,may be to place a mobile order for coffee. For example, the quick actionmay allow a user to order and pay for their order without ever enteringthe point of interest. The third most relevant point of interest may bea bowling alley, shown as place card 944. A quick action for the bowlingalley may allow a user to reserve a bowling lane.

The order of which the places of interest appear may change based on theaudio properties searched by the user. The order of which the places ofinterest appear may be based on the distance from the location of thedevice or a preselected geographical location chosen by the user. Insome examples, the order may also change as the system learns thechoices of the users.

FIG. 10 illustrates an example of a display showing an expanded placecard. Device 1000 may be similar to device 900. For example, device 1000may include a display 1020, output 1016, input 1010 and at least onemicrophone 1012. These features may be similar to those discussed hereinwith respect to device 100.

Expanded place card 1062 may be displayed once a user selects a point ofinterest from the search results. For example, the user may select aninteractive place card from the search results to find out more detailedinformation for that point of interest. For example, FIG. 10 illustratesan expanded place card 1062 for the bowling alley 1044. The expandedplace card 1062 may include an address 1050, operating hours 1052, aphone number 1054, and a website 1056 associated with the bowling alley.

The expanded place card 1062 may further include audio propertiesassociated with the bowling alley 1044. For example, details regardingthe type of music, or music genre 1058, that is played at the point ofinterest may be displayed. The music genre 1058 may be broken down basedon the day of the week. As shown in FIG. 10, the bowling alley 1044 mayplay country music on Wednesdays, rock and roll music on Thursday, popmusic on Fridays, and live music on Saturdays. The music genres 1058 mayupdate as audio properties are collected and verified by users. Forexample, the music genre 1058 may change such that it lists that thereis a DJ on Thursday nights instead of rock and roll music. Thus, themusic genres 1058 are not static but, instead, updated in real timebased on audio properties determined by user devices at that point ofinterest.

The expanded place card 1062 may further include a volume level section.Volume level 1060 may include an indication of how loud the backgroundvolume level is at a current time. In this example, typical volumelevels are indicated by unshaded bars 1064, while the current volumelevel is indicated by shaded bar 1066. In this regard, the currentvolume is shown in relation to the typical or historical volume level,thereby providing a frame of reference. A timeline 1068 may also beprovided for reference, indicating a time corresponding to each bar inthe chart. Each bar may correspond to a one hour interval, or to anyother interval of time. Moreover, the timeline 1068 may correspond to arange of hours that the point of interest is in operation. While thevolume levels are shown as being displayed in a bar graph, it should beunderstood that this is merely one example. The volume levels may beprovided for display in any number of ways, such as text, pictorialdiagrams, charts, graphs, etc.

The volume level 1060 section of expanded place card 1062 may assist auser on determining whether the point of interest, bowling alley 1044,is appropriate for their visit. For example, people with hearing lossmay want to visit during a time where the volume level is low. In someexamples, if a user is seeking a place to go with a group of friends,the user may not mind if the volume level is louder.

The volume level 1060 section of expanded place card may further includea break down as to what contributes to the volume level. For example,the volume level 1060 may be made up of conversations and music. In someexamples, there may be an indication of the volume level of theconversations and an indication of the volume level of the musicoccurring at the point of interest.

FIG. 11 illustrates an example method for accessing a database havingthe information pertaining to a plurality of points of interest and theassociated audio properties. For example, in block 1110 the device mayreceive a search query. The search query may be performed by one or moreprocessors that are part of the network 250, as described herein. Thesearch query may be for one or more points of interest. The search querymay include one or more audio properties associated with the points ofinterest. According to some examples, the search query may includelocation information. In some examples, the location information may beobtained based on the location of the device. According to someexamples, the location information may be a location selected by theuser. The location may be determined based on a city and state, the zipcode, an address, etc. The search query may include search attributespertaining to audio properties associated with the points of interest.For example, the search query may be for a quiet cafè that allows forconversation or a bar that plays country music.

In block 1120, the device and/or one or more processors that are part ofthe network 250 may access the database of a plurality of points ofinterest. The database may further include information associated withthe plurality of points of interest. For example, the database mayinclude information pertaining to the location of the points ofinterest, operating hours of the points of interest, the phone number ofthe points of interest, the website of the points of interest, the typeof music such as musical genres played at the points of interest, thevolume level at the points of interest, etc. According to some examples,one or more selected points of interest may be identified. The audioproperties associated with the selected points of interest maycorrespond with the audio properties included in the search query.

In block 1130, the device and/or one or more processors that are part ofthe network 250 may filter through the plurality of points of interestwithin the database based on the location information and audioproperties provided in the search query. For example, if the usersearches for a quiet restaurant within a preset distance from thelocation of the device, the device and/or one or more processors thatare part of the network 250 may filter through the database to find onlyrestaurants that are quiet and within the preset distance. In someexamples, the user may search for a place having activities for kidsthat plays music appropriate for children. According to some examples,the device and/or one or more processors that are part of the network250 may filter the one or more points of interest based on locationinformation received as part of the search query.

In block 1140, the device and/or one or more processors that are part ofthe network 250 may provide information indicating one or more of thepoints of interest that fulfill the criteria set in the search query.The information may be provided for display on the device. In someexamples, the information may be provided as audio output.

As will be appreciated, the technology described herein may enable asearch for points of interest based on audio properties associated withthat point of interest. The audio properties may be, for example, thebackground noise volume, music volume, conversation volume, music genre,etc. Users with high frequency hearing loss may search for points ofinterest having a low or quiet background noise volume. In someexamples, people with reduced hearing who are looking to hear music maysearch for points of interest having a loud music volume. Allowing usersto identify a plurality of points of interest based on associated audioproperties may benefit the user in selecting a particular point ofinterest best suited to their interest.

Unless otherwise stated, the foregoing alternative examples are notmutually exclusive, but may be implemented in various combinations toachieve unique advantages. As these and other variations andcombinations of the features discussed above can be utilized withoutdeparting from the subject matter defined by the claims, the foregoingdescription of the embodiments should be taken by way of illustrationrather than by way of limitation of the subject matter defined by theclaims. In addition, the provision of the examples described herein, aswell as clauses phrased as “such as,” “including” and the like, shouldnot be interpreted as limiting the subject matter of the claims to thespecific examples; rather, the examples are intended to illustrate onlyone of many possible embodiments. Further, the same reference numbers indifferent drawings can identify the same or similar elements.

1. A system, comprising: one or more memories storing a plurality ofpoints of interest in association with one or more audio properties; oneor more processors in communication with the one or more memories, theone or more processors configured to: receive, from at least one clientcomputing device, a search query for one or more points of interest, thesearch query including one or more audio properties associated with theone or more points of interest; identify, in response to the searchquery, one or more selected points of interest stored in the one or morememories, wherein the audio properties associated with the one or moreselected points of interest correspond with the one or more audioproperties included in the search query; and provide for output at leastone of the selected points of interest along with the corresponding oneor more audio properties.
 2. The system of claim 1, wherein the one ormore processors are further configured to: receive location informationassociated with the search query; and wherein identifying the one ormore selected points of interest stored in the one or more memoriesfurther comprises filtering the one or more selected points of interestbased on the received location information.
 3. The system of claim 1,wherein the one or more audio properties include at least one of a musicgenre, a title of currently playing song, or a volume level.
 4. Thesystem of claim 3, wherein the volume level indicates a music playbackvolume or a background noise volume.
 5. The system of claim 1, whereinthe plurality of points of interest in association with one or moreaudio properties stored in the one or more memories are updated inreal-time.
 6. The system of claim 1, wherein the one or more memoriescomprise a mapping database including the one or more points ofinterest, wherein the one or more audio properties are stored in themapping database in association with the plurality of points ofinterest.
 7. The computing device of claim 1, wherein the output for theat least one of the selected points of interest indicates at least oneof a music genre and a volume level.
 8. A method comprising: receiving,from at least one client computing device, a search query for one ormore points of interest, the search query including one or more audioproperties associated with the points of interest; identifying, inresponse to the search query, one or more selected points of interest,wherein the audio properties associated with the one or more selectedpoints of interest correspond with the one or more audio propertiesincluded in the search query; and providing for output at least one ofthe selected points of interest along with the corresponding one or moreaudio properties.
 9. The method of claim 8, further comprising:receiving location information associated with the search query; andwherein identifying the one or more selected points of interest furthercomprises filtering, based on the received location information, the oneor more selected points of interest.
 10. The method of claim 8, whereinthe one or more audio properties includes at least one of a music genre,a title of a currently playing song, or a volume level.
 11. The methodof claim 10, wherein the volume level indicates a music playback volumeor a background noise volume.
 12. The method of claim 8, furthercomprising updating, in real-time, the plurality of points of interestin association with the one or more audio properties.
 13. The method ofclaim 8, further comprising storing the one or more audio properties inassociation with the plurality of points of interest in a mappingdatabase.
 14. The method of claim 8, wherein providing for output atleast one of the selected points of interest further comprisesindicating at least one of a music genre and a volume level.
 15. Anon-transitory computer readable storage medium storing instructionsexecutable by a processor for performing a method comprising: receiving,from at least one client computing device, a search query for one ormore points of interest, the search query including one or more audioproperties associated with the one or more points of interest;identifying, by one or more processors in response to the search query,one or more selected points of interest, wherein the audio propertiesassociated with the one or more selected points of interest correspondwith the one or more audio properties included in the search query; andproviding, by the one or more processors, for output at least one of theselected points of interest along with the corresponding one or moreaudio properties.
 16. The non-transitory computer readable medium ofclaim 15, further comprising: receiving, by the one or more processors,location information associated with the search query; and whereinidentifying the one or more selected points of interest responsive tothe search query further comprises filtering, by the one or moreprocessors based on the received location information, the one or moreselected points of interest.
 17. The non-transitory computer readablemedium of claim 15, wherein the one or more audio properties includes atleast one of a music genre, a title of a currently playing song, or avolume level.
 18. The non-transitory computer readable medium of claim17, wherein the volume level indicates a music playback volume or abackground noise volume.
 19. The non-transitory computer readable mediumof claim 15, further comprising updating, by the one or more processors,in real-time the plurality of points of interest in association with theone or more audio properties.
 20. The non-transitory computer readablemedium of claim 15, further comprising storing the one or more audioproperties in association with the plurality of points of interest in amapping database.